Papers by Karthik R Narasimhan
An Annotated Dataset of Errors in Premodern Greek and Baselines for Detecting Them (2025.findings-naacl)
Copied to clipboard
Creston Brooks, Johannes Haubold, Charlie Cowen-Breen, Jay White, Desmond DeVaul, Frederick Riemenschneider, Karthik R Narasimhan, Barbara Graziosi
| Challenge: | a dataset of real errors in premodern Greek is presented to improve error detection methods . scribal errors are more difficult to detect than print or digitization errors. |
| Approach: | They propose to annotate 1,000 words more likely to contain errors and annotated them as errors or not . they propose to evaluate new error detection methods that outperform other methods . |
| Outcome: | The proposed method outperforms all other methods, improving true positive rate by 5%. |
Contextual Experience Replay for Self-Improvement of Language Agents (2025.acl-long)
Copied to clipboard
| Challenge: | Existing large language model (LLM) agents fail in complex tasks without any environment-specific experiences. |
| Approach: | They propose a framework that accumulates and synthesizes past experiences into a dynamic memory buffer to enable efficient self-improvement for language agents in their context window. |
| Outcome: | The proposed framework improves performance on WebArena and VisualWebAren . it surpasses tree search method with fewer token costs and achieves state-of-the-art performance of 31.9%. |
PersonaGym: Evaluating Persona Agents and LLMs (2025.findings-emnlp)
Copied to clipboard
Vinay Samuel, Henry Peng Zou, Yue Zhou, Shreyas Chaudhari, Ashwin Kalyan, Tanmay Rajpurohit, Ameet Deshpande, Karthik R Narasimhan, Vishvak Murahari
| Challenge: | Persona agents are LLM agents conditioned to act according to an assigned persona . evaluating how faithfully these agents adhere to their personas remains a challenge . |
| Approach: | a new study evaluates persona agents' ability to act according to an assigned persona . a persona agent's person score is a human-aligned automatic metric that can be used to evaluate a model . |
| Outcome: | a new evaluation framework and a human-aligned automatic metric show that persona agents can perform better. |